07. Applying Reinforcement Learning in Trading

Overview of Q-Learning in Reinforcement Learning

• Understanding Policy is essential for training an agent effectively:

  • Stochastic vs. Deterministic Policy: We will use stochastic for flexibility.
  • On-Policy Learning: Both target and behaviour policies are the same.
  • Off-Policy Learning: Different target and behaviour policies, offering more exploration.

• Q-Learning is a popular off-policy reinforcement learning method:

  • Focuses on maximizing cumulative reward without needing a model of the underlying environment.
  • Utilizes a Q-table to store expected rewards for state-action pairs.
  • Employs a neural network, or Deep Q Network (DQN), to approximate the Q-table.

• Key Components and Concepts:

  • Bellman Equation: Updates Q-values based on rewards and future state predictions.
  • Neural Network: Optimized via. gradient descent to approximate optimal Q-values.
  • Exploration vs. Exploitation: Balance between exploring new actions and choosing known profitable actions - managed through strategies such as ε-greedy.

Focus on these core elements to understand how Q-learning will build effective trading algorithms.

Supplementary Material - On- vs. Off-Policy Learning

The Following Medium article gives a good description of on- and off-policy learning. It also gees through a different strategy for Off-Policy learning called "Important Sampling". This will not be used in this course, as we use Q-learning as our off-policy learning technique.

On-Policy v/s Off-Policy Learning

Supplementary Material - Exploration, Exploitation, and ε-greedy Learning

The following Medium article gives a deep-dive into the exploration-exploitation dilemma and how the ε-greedy strategy is used to address this dilemma.

RL Series#3: To explore or not to explore, that is the question